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LONG-TERM  GOALS 

We  are  developing  a  high-performance  broadband  radiation  code  for  the  current  generation  of 
computational  architectures.  This  code,  called  RRTMGP,  will  be  a  completely  restructured  and 
modem  version  of  the  accurate  RRTMG  radiation  code  {Mlawer  et  ah,  1997;  lacono  et  ah,  2008)  that 
has  been  implemented  in  many  General  Circulation  Models  (GCMs)  including  the  Navy  Global 
Environmental  Model  (NAVGEM),  the  NCAR  Community  Earth  System  Model  (CESM),  and 
NOAA’s  Global  Forecast  System  (GFS).  Our  proposed  development  will  significantly  lessen  a  key 
bottleneck  in  these  highly  complex  and  coupled  models,  namely  the  large  fraction  of  computational 
time  currently  required  for  the  calculation  of  radiative  fluxes  and  heating  rates. 

OBJECTIVES 

The  radiation  calculations  needed  for  climate  simulations  require  many  independent  and  complicated 
calculations,  and  are  therefore  an  inviting  target  for  new  computing  architectures  such  as  Many- 
Integrated-Cores  (MICs)  and  Graphical  Processing  Units  (GPUs).  We  are  developing  RRTMGP  (‘P’ 
stands  for  ‘parallel’),  a  modern  version  of  the  radiation  code  (RRTMG)  used  by  many  climate  models, 
directed  at  the  current  generation  of  vector-  and  cache-based  computational  architectures.  This  code 
will  retain  the  high  accuracy  of  RRTMG,  but  is  being  developed  from  scratch  to  make  it  more  flexible 
and  amenable  to  optimization  across  a  wide  range  of  platforms.  The  objective  is  a  single  well- 
maintained,  well-documented,  and  efficient  radiation  code  that  can  be  used  by  the  modeling 
community  for  a  diverse  range  of  applications  across  a  wide  range  of  computing  facilities.  RRTMGP 
will  exhibit  profound  improvements  in  speed  for  GPU  and  vector  CPU  machines  and  lesser,  but  still 
valuable,  speed-ups  on  other  CPU-based  platforms  relative  to  the  current  version  of  the  code. 

APPROACH 

The  collaborating  team  consists  of  scientists  and  programmers  with  detailed  knowledge  of  RRTMG 
and  its  use  within  GCMs,  as  well  as  representatives  of  modeling  centers  that  use  RRTMG  and  plan  to 
upgrade  to  RRTMGP.  Eli  Mlawer  of  AER,  the  lead  developer  of  RRTMG,  is  the  PI  of  this  project, 
and  his  team  at  AER  includes  programmers  with  experience  coding  for  modern  computer  architectures, 
including  the  recent  GPU-based  implementation  of  RRTMG.  Team  member  Robert  Pincus  of 
University  of  Colarado  refactored  RRTMG  to  create  PSRad,  which  has  served  as  a  prototype  for  the 
current  development.  Drs.  Mlawer  and  Pincus  are  leading  the  design  and  development  of  RRTMGP. 
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Brian  Eaton  of  NCAR,  which  has  employed  RRTMG  in  its  GCM  (CESM/CAM)  for  nearly  a  decade, 
leads  a  group  at  this  modeling  center  participating  in  the  project.  In  the  initial  phase  of  the  project,  the 
NCAR  focus  has  been  on  profiling  the  performance  of  RRTMG  as  used  in  their  GCM.  Project 
collaborators  Ming  Eiu  and  Tim  Whitcomb  of  NRE  represent  the  interests  of  the  Navy  GCM 
(NAVGEM). 

Due  to  the  expected  wide  impact  of  this  development  effort  on  climate  and  weather  modeling, 
representatives  from  a  number  of  different  modeling  centers  have  contributed  their  perspectives  on  this 
development  effort.  This  includes  John  Michalakes  (NOAA),  Robin  Hogan  (ECMWE),  a  number  of 
colleagues  at  the  Max-Planck  Institute,  and  Will  Sawyer  and  Marcus  Wetzstein  (Swiss  Supercomputer 
Center). 

WORK  COMPLETED 

We  completed  a  working  version  of  the  new  longwave  code,  RRTMGP_EW_vO.  This  version  is 
capable  of  computing  fluxes  and  heating  rates  and  has  the  anticipated  final  organization  and  structure 
of  RRTMGP.  RRTMGP  is  not  yet  ready  for  scientific  application,  partly  because  some  aspects  (e.g. 
McICA  sampling  of  cloud  variability)  is  not  yet  implemented  but,  more  importantly,  because  the 
underlying  spectroscopic  information  is  somewhat  out  of  date,  so  that  the  accuracy  of  our  simplified 
algorithms  can  not  be  thoroughly  evaluated.  RRTMGP_EW_vO  has  been  provided  to  our  NASA 
GSEC  collaborators  and  NCAR  colleagues  for  evaluation  and  profiling,  and  colleagues  in  the  Swiss 
climate  modeling  center  will  adapt  this  code  to  run  on  a  GPU  for  implementation  into  their  GCM 
(based  on  the  Max  Planck  Institutes’s  ICON  model). 

A  completely  new  driver  has  been  developed  for  this  version,  interfacing  with  the  calling  GCM  with 
required  (for  each  atmospheric  column)  inputs:  layer  average  temperatures,  layer  average  pressures, 
layer  edge  pressures,  surface  skin  temperature,  and  gas  concentrations  (derived  type)  for  all  modeled 
gases.  Optional  inputs  are  the  layer  column  amounts  of  dry  air  and  layer  edge  temperatures.  A  fully 
functional  gas  optics  module  has  been  developed;  in  addition  to  its  code,  this  module  contains  an 
object  class  that  specifies  the  spectral  bands,  their  g-points,  all  stored  coefficients,  etc.  This  module 
has  these  same  inputs,  and  its  outputs  are  (per  g-point  and  atmospheric  column)  gas  optical  depths,  the 
internal  (i.e.  Planck)  layer  source  irradiances,  the  internal  source  irradiances  for  increasing  (relative  to 
the  layer  ordering  provide  by  the  user)  layer  edges,  the  internal  source  irradiances  for  decreasing  layer 
edges,  and  the  source  irradiances  from  the  surface.  The  gas  optical  depth  calculation  includes 
contributions  from  major  species,  minor  species  (though  not  all  have  stored  coefficients  yet),  and  the 
water  vapor  foreign  and  self  continua.  The  interpolation  scheme  needed  to  compute  absorption 
coefficients  from  gas  concentrations,  temperature,  and  pressure  has  been  completely  redesigned 
compared  to  RRTMG.  Eor  this  development,  we  increased  (compared  to  RRTMG)  the  high  range  of 
surface  pressures  to  1100  mb  so  that  we  encompass  the  highest  sea  level  pressure  recorded  below  750 
m  (1083.8  mbar).  Likewise  we’ve  expanded  our  range  of  Earth-based  temperatures  to  160  -  355  K  so 
our  absorption  coefficients  are  valid  for  more  extreme  cases.  Making  these  changes  in  RRTMGP 
required  changes  to  our  systems  and  scripts  that  generate  the  stored  k-distributions.  We’ve  now  built 
flexibility  into  our  updated  scripts  to  facilitate  future  modifications  of  the  minimum/maximum 
temperatures/pressures  in  the  code. 

A  new  longwave  absorption-emission  solver  has  been  developed,  with  exact  treatment  of  the  linear-in- 
tau  approximation.  Although  there  is  not  yet  a  solver  for  performing  longwave  calculations  that 
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include  scattering,  all  components  needed  for  these  calculations  ready  and  the  solver  can  be  dropped 
into  place. 

Since  the  solver  operates  on  each  g-point  independently  while  the  gas  optics  code  operates  most 
efficiently  operates  with  g-point  as  its  inner  loop,  at  the  gas  optics  routine  reverses  the  index  order  of 
its  input  before  performing  its  calculations  and  then  reverses  the  order  of  its  output  after  its 
calculations  are  performed. 

All  output  from  the  gas  optics  module,  including  the  components  of  the  optical  depth  (e.g.  major, 
continuum),  has  been  validated  with  respect  to  the  corresponding  output  from  the  development  code, 
which  has  the  same  coding  structure  as  in  RRTMG.  This  validation  exercise  was  successfully 
performed  for  layering  that  goes  from  surface  to  TOA,  as  well  as  TOA  to  surface.  A  script  has  been 
developed  that  calls  the  driver  using  input  in  netcdf  format,  allowing  validation  of  the  calculated  flux 
and  cooling  rates. 

A  new  build  system  has  been  developed  for  easier  integration  into  existing  projects.  It  supports  various 
Fortran  compilers,  automatically  determines  and  displays  dependencies,  and  allows  for  a  parallel  build. 

In  addition,  Co-PIs  Pincus  and  Wehe  attended  the  ESPC  AOLI  meeting  in  Boulder,  CO,  in  November 
2014,  which  was  hosted  by  Pincus.  At  this  meeting,  Pincus  presented  an  overview  of  our  project’s 
goals  and  status.  (PI  Mlawer  could  not  attend  due  to  a  scheduling  conflict  with  another  meeting.) 

RESULTS 

The  release  of  RRTMGP_LW_vO  represents  a  significant  milestone  for  the  project.  The  developed 
code  strikes  a  balance  between  readability  and  comprehensibility  (for  its  anticipated  scientific  user 
base)  and  advanced  computational  features  and  flexibility  (critical  for  parallelization).  RRTMGP_vO 
makes  extensive  use  of  Fortran  2003  classes  and  abstract  interfaces.  Input  classes  have  been 
developed  for  gas  concentrations,  random  number  generators  (needed  for  cloud  sampling  to  treat 
varaibility),  cloud  properties,  and  aerosol  properties.  Computed  fluxes  constitute  another  class.  This 
structure  allows  simplification  of  run-time  choices,  such  as  the  number  of  streams  used  in  the  flux 
calculation  and  the  spectral  integration  of  the  output  fluxes.  Code  efficiency  is  gained  from  exposure 
of  fine-grained  parallelism  and  algorithmic  simplification,  especially  a  reduction  in  branching.  The 
code  and  the  stored  data  it  utilizes  are  independent  of  each  other;  the  data  is  provided  at  run  time. 

Given  the  modular  nature  of  this  code,  it  will  be  relatively  straightforward  to  modify  to  run  in  the 
shortwave  region.  The  shortwave  code  will  reuse  most  of  the  longwave  components;  the  main 
difference  will  be  the  data  it  utilizes. 

The  build  system  that  has  been  developed  will  allow  for  easier  and  more  flexible  integration  over  a 
wide  range  of  computing  architectures. 

Development  activities  at  NCAR 

Our  NCAR  collaborators,  led  by  Brian  Eaton,  reviewed  the  proposed  RRTMGP  design  and  provided 
useful  feedback.  In  addition,  they  ran  the  RRTMGP  build/test  framework  on  NCAR  compute 
platforms  and  provided  feedback.  They  also  began  the  task  of  refactoring  the  top-level  radiation 
interface  in  CAM  to  prepare  for  the  inclusion  of  RRTMGP. 


3 


IMPACT/APPLICATIONS 


The  development  of  RRTMGP  should  have  a  significant  impact  on  the  ability  of  GCMs,  inclduing 
NAVGEM  and  CESM,  to  perform  efficient  and  accurate  simulations  of  climate  and  weather. 

TRANSITIONS 

The  initial  version  of  the  high-performance  broadband  longwave  radiation  code  we  are  developing 
under  this  support,  RRTMGP_EW_vO,  has  been  completed  and  distributed  to  selected  colleagues  at 
modeling  centers,  including  NOAA,  NCAR,  and  CSCS.  Our  colleagues  are  analyzing  the  new  code’s 
structure  for  a  number  of  different  compilers,  and  continue  to  provide  us  with  useful  feedback.  They 
will  also  be  profiling  the  code’s  computational  performance. 

RELATED  PROJECTS 

In  addition  to  their  analysis  of  RRTMGP_EW_vO,  our  colleagues  at  the  Swiss  Supercomputer  Center 
(CSCS)  in  Eugano  will  be  developing  a  GPU  version  (OpenACC)  of  this  code  for  use  in  the  ICON 
EES  model.  This  version  will  provide  a  significant  foundation  for  the  GPU  version  of  our  code  that  is 
a  deliverable  for  this  project.  Andre  Wehe  of  AER  will  spend  the  first  week  in  November  in  Eugano  to 
both  assist  Will  Sawyer  and  his  team  with  this  development  and  to  be  tutored  in  GPU  programming 
techniques  by  CSCS  experts.  This  will  support  our  future  ability  to  develop  and  maintain  GPU 
versions  of  the  high-performance  radiation  code. 
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